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ABSTRACT 

Research conducted to determine the effect of certain 
measurable characteristics of written material upon speed of 
memorization is presented. The characteristics studied fall into four 
classes: (1) Load measures reflecting informational density, (2) 
Length measures based upon number of syllables, words, kernel 
sentences, clauses, or sentences in a passage, (3) Packaging measures 
based on alternative, gramatically equivalent ways for dealing with 
the same semantic material, and (4) Word frequency measures. Six 
studies comprising 14 experiments were conducted. Experimental 
materials consisted of paragraphs, lists of sentences, single 
sentences, and lists of words. In all but one of the six studies, the 
primary measure of learning was cumulative presentation time to a 
criterion of one perfect rote recall of the experimental material, 
where presentation time during each exposure of the material was 
under the subject's control. Results include: (1) Paragraphs with a 

very high ratio between content words and total words in the 
paragraph required more time to learn than those with a lower content 
load; and (2) The time required to learn a set of sentences increased 
as the mean number of syllables per content word increased. 
Conclusions include: (1) Of the four types of stimulus 

characteristics studied, length has the most potent effects upon rote 
memorization; and (2) A number of relationships between independent 
and dependent variables were found to yield good fits to linear, 
power, or exponential equations. (Author/CK) 
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FOREWORD 



The studies described in this report were undertaken as part of the 
research conducted by the Human Resources Research Office under 
Basic Research Study 7, Precision of Statement and Perception of 
Meaning of Written Language in Military Training. These studies relate 
to the problem of quantifying the difficulty of written material on tech- 
nical subjects and determining empirically rules for writing that mini- 
mize the difficulty of such material. 

Basic Research Study 7 was conducted by HumRRO Division No. 5 
(Air Defense) at Fort Bliss, Texas, while Dr. Robert D. Baldwin was 
Director of Research. Military support for the study was provided by 
the U.S. Army Air Defense Human Research Unit. The Military Chief 
of the Unit was LTC Leo M. Blanchett, Jr. SP 4 Fred G. Hampton pro- 
vided assistance in the data collection and analysis phases of the research. 

HumRRO research is conducted under Army Contract DA 44-188-ARO- 2, 

with the Basic Research studies under Army Project No. 2J014501B74B 02, 
Basic Research in Psychology and Social Science. 



Meredith P. Crawford 
Director 

Human Resources Research Office 
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GLOSSARY 



Content Word Ratio (CWR) 
Content words 

Free recall 
Function words 
Kernel sentence 



Length 

Linear relationship 
Load 

Monotonic relationship 



Packaging 
Rote recall 

Tokens 

Trial 

"Trial" 

Types 

Type-Token Ratio (TTR) 
Word frequency 



The ratio between total content words and total words in the passage. 

Words commonly classified as nouns, verbs, adjectives, and adverbs, 
together with substitute classes of words (such as pronoun for noun). 

A response in which elements may occur in any order. 

Words not classified as content words. 

A grammatical unit that can be applied to text to analyze content in 
terms of simplest possible declarative statements that express com- 
plete thoughts; that is, in each instance, anything less becomes a 
non-sentence. A normal sentence is comprised of one or more ker- 
nel sentences. 

Number of sentences, clauses, kernel sentences, words and syllables 
have all been used as measures of length. 

Form of relationship between two variables that can be represented 
by a straight line (or the equation for a straight line: Y = aX +b), 
where X and Y are variables and a and b are constants. 

An index of information; the information density of a unit of content 
of fixed length. 

Form of relationship in which direction of change in value in one 
variable is associated with uniformity of direction of change for the 
other. That is, the value for one variable (either increasing or 
decreasing) is associated with a uniform direction of change in 
the other (either increasing or decreasing) over the range of both 
variables. A linear relationship is the simplest form of a monotonic 
relationship. A relation in which there are cyclic changes in a varia- 
ble with increasing values of another variable illustrates functions 
that are not monotonic. 

Alternative construction for dealing with the same semantic material. 

A response in which elements must occur in the order in which they 
occur in the stimulus. 

All words in a passage (see Type-Token Ratio). 

Fixed-length exposure of a unit of content. 

Variable-length (subject-controlled) exposure of a unit of content. 

Different words in a passage (see Type-Token Ratio). 

The ratio of the number of different words (types) to this total number 
of words (tokens) in a passage. 

A word's frequency in print, as reported by one of the counts in 
Reference 18. 




Problem 

The research conducted under Basic Research Study 7 of the Human Resources Research 
Office bears upon two related problems. The first problem is that of determining how to evaluate 
the difficulty of technical written material; for any example of written material dealing with a 
technical subject, it would be desirable to be able, eventually, to assign a number to the material 
which would indicate where it falls on a scale of difficulty. The second problem is that of 
determining what can be done to make given material less difficult. If a writer has two or more 
options as to how he can express a proposition, idea, relationship, and so forth, which option 
should he elect in order to minimize the difficulty a reader will experience in trying to under- 
stand the writing? 

The present studies were conducted to determine the effect of certain measurable charac- 
teristics of written material upon speed of memorization. 1 While memorizing and understanding 
are not the same process, memorization measure s-which are quicker, more reliable, and easier 
to use than measures of understanding-were used in these experiments on the assumption that 
such measures would be predictive of understanding. 

The characteristics studied in these experiments fall into four general classes: 

(1) Load measures reflecting informational density. 

(2) Length measures based upon number of syllables, words, kernel sentences, 
clauses, or sentences in a passage. 

(3) Packaging measures based on alternative, grammatically equivalent ways for 
dealing with the same semantic material. 

(4) Word frequency (familiarity) measures. 

Research Procedure 

Six studies comprising 14 experiments were conducted. Experimental materials consisted 
of paragraphs, lists of sentences, single sentences, and lists of words. 

In all but one of the six studies, the primary measure- of learning was cumulative presen- 
tation time to a criterion of one perfect rote recall of the experimental material, where presen- 
tation time during each exposure of the material was under the subject's control. In the sixth 
study, three experiments employed the same measure, but to a criterion of one perfect free recall 
of the material. In two other experiments, exposure time on each trial was fixed and thus 
not under the subject's control; a trials-to-criterion measure of learning speed was used in 
these experiments. 

Results 

(1) Paragraphs with a very high ratio between content words (nouns, verbs, adjectives, 
and adverbs) and total words in the paragraph required more time to learn than those with a lower 
content word load. However, content word load variations within or below the normal range for 

written English had negligible effects on learning time. 

(2) The time required to learn a set of sentences increased as the mean number of syl- 
lables per content wold increased. 




‘As a convenience to the reader, a glossary of some technical, linguistic, and mathematical terms 
defined as they are used in this report, is presented on p. lv. 
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(3) In two studies using small samples of subjects, reasonably smooth curves were found 
to represent the effects of list length, in sentences, upon recall. Learning time measures were 
differently related to list length in the two studies, apparently due to the fact that the amount 
of content word repetition varied with list length in one study and not in the other. 

(4) A pilot study yielded results which suggest that single-sentence recall is an orderly 
joint function of number of kernel sentences in the clause and number of clauses in the sentence. 

(5) In material featuring a string of serial noun phrase modifiers, placing the series before 
the noun phrase head, rather than after it, was found to facilitate single-sentence recall. When 
modifiers were coordinate, however, learning was about the same regardless of whether the 
modifiers were placed before or after the head. Where pre-head placement led to quicker learning, 
the result can be accounted for entirely on the basis of sentence length (in syllables), since 
the post-head position of the modifiers required use of more wards in the sentence. 

(6) Studies on the joint effects of word length and word frequency on the recall of word 
lists indicate that presentation time to criterion is a monotonic, increasing function of word 
length. Simple frequency effects and joint effects of length and frequency varied from one study 
to another with changes in procedure and in the particular lists used. In all such studies, mean 
response rate increased with increasing word frequency. 

Conclusions 

(1) Of the four types of stimulus characteristics studied— load, length, packaging, and 
frequency— length has by far the clearest and most potent effects upon rote memorization. 

(2) A number of relationships between independent and dependent variables were found 
to yield good fits to linear, power, or exponential equations. Results of this type may have 
application in the eventual development of mathematical equations that will represent and meas- 
ure the level of difficulty of technical written material. 

(3) The present findings have implications for the possible course of future research 
exploring the nature of technical language difficulty. Some of these implications are discussed 
in Chapter 8. 
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Effects of Grammatical Factors 
and Amount of Material 
Memorizing Paragraphs, Sentences, 
and Word Lists 



Chapter 1 

INTRODUCTION 



This report describes a series of experimental studies, conducted in 
Basic Research Study 7 of the Human Resources Research Office, exploring 
language factors that may influence the difficulty of technical written material. 
The general objectives of the research in this area are: 

(1) To predict the effort that a reader will need to expend in order to 
understand given written material. (How difficult is the material?) 

(2) To indicate how such material can be modified to minimize its 
superficial complexity so that the reader will need to expend l§ss effort to 
understand it. (What can be done to make the material less difficult?) 



BACKGROUND 

The research stems from two primitive, or basic, notions. First, written 
material has measurable characteristics that determine its level of complexity. 
Second, the level of complexity of a passage determines how difficult it will be 
for a given reader to understand. Studies predicated on these notions have been 
under way in one form or another for 40 years. The earliest of these investi- 
gations can be traced to the practical requirement in public school education 
that textbooks be written so school children at particular grade levels can 
understand them. 

At least three trends in educational and psychological research appear in 
some degree relevant to the present work. The earliest studies sought to pre- 
dict the readability of written material on the basis of correlations between 
reading test scores and certain gross statistical and grammatical character- 
istics of the material. Such studies, leading to the production of readability 
formulas, appear with regularity in the literature. 

The studies reflecting a second trend— experimental rather than correla- 
tional in approach— seek to predict the understandability of written material in 
consequence of deliberate manipulation of certain characteristics of the material. 
There ari fewer studies of this sort. Most of them are recent and, as a result, 
some of them make use of recent advances in the theory of grammar to define 
characteristics of the material which will be manipulated (a tendency not yet 
evident in the readability research). 

A third research trend relates to language development in children. Studies 
on this topic traditionally have addressed the proposition that as children 
mature, their writing will become linguistically more complex. A recent tend- 
ency in language development research has been to focus on the grammatical 
complexity of the unit of reading and its relation to developmental levels. 

While we do not intend here to review the literature associated with these 
trends, perhaps it would be useful to describe their methodologies. 

Correlational Research . Correlational research has tended almost 
exclusively to be readability research. The readability formula is a device 
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used by educators to evaluate the relative difficulty of textbooks. These for- 
mulas typically are regression equations based upon correlation studies. At the 
outset of such a study, a series of passages of given length is evaluated for 
several characteristics, such as average length of sentences, number of depend- 
ent clauses, number of prepositional phrases, or number of repeated words. 
Then, subjects read the passages and are tested for understanding. Next, test 
scores are correlated with the various passage characteristics. Finally, those 
characteristics which correlate most highly with test scores (and not too highly 
with each other) are used as factors in readability formulas. 

The readability research has been reviewed extensively (1, 2, 3,4). We 
will be content here to give an indication of the kinds of factors employed in 
readability formulas. 

Perhaps the most comprehensive research of this sort was reported 
30 years ago by Gray and Leary (5). The effects of approximately 70 charac- 
teristics of written material upon the understanding of adults of limited reading 
ability were determined by correlational methods. A series of alternative 
readability formulas was generated on the basis of the resulting matrix of 
intercorrelations. According to the principal formula of Gray and Leary, test 
score is a function of five factors: 

(1) Number of personal pronouns in a 100-word passage. 

(2) Number of different difficult words in the passage. 

(3) Average length of sentences. 

(4) Percentage of different words. 

(5) Number of prepositional phrases. 

The household word for readability research is Flesch. According to 
the Flesch reading ease formula (6), reading ease score is a function of: 

(1) Average sentence length of the passage, in words. 

(2) Number of syllables per 100 words. 

The economy, in factors used, of the Flesch formula over the Gray 
and Leary formula is noteworthy. The Flesch formula reflects a tendency in 
readability research in recent years to yield formulas based on only a few fac- 
tors, of the sort that can be reliably assessed by almost any literate person. 
Perhaps such simple formulas are adequate when used to classify textbooks by 
age-grade at the public school level. Whether they are adequate for evaluating 
the difficulty of technical material of the sort found in college-level introductory 
textbooks or in the course materials of Army Technical Training programs is 
a question that has received only limited study. 

Stevens and Stone (7) applied one of the Flesch formulas to textbooks 
in psychology and found, according to the formula, that the writings of the 
psychologist Koffka were among the easiest readings in the field. However, 
most psychologists would agree that Koffka' s writings are monumentally diffi- 
cult. We have concluded— on the basis of informal, intuitive studies like those 
of Stevens and Stone — that readability formulas are not sufficiently sensitive 
to establish the relative difficulty of technical materials written for use in 
adult education. 

Experimental Research . While experimental and correlational approaches 
to evaluating the difficulty of written material may differ methodologically in 
a number of ways, they need so differ in only one. The passages of an experi- 
mental study are deliberately written to reflect different levels of a few charac- 
teristics of written material— one characteristic if the research design is 
simple, more than one if it is complex. All other characteristics of the experi- 
mental passages are held constant. The passages of a correlational study are 
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not deliberately written by the investigator. If they were, no doubt they would 
reflect the same research design considerations to which the experimental 
investigator reacts. Under the latter condition, results of parallel investiga- 
tions employing the two approaches would amount to no more than two different 
ways of saying the same thing. 

The passages of a correlational study typically are selected from 
existing written material. The selection process guarantees that the set of 
passages used will reflect different levels for each of a number of character- 
istics of written material. However, while the experimental study allows the 
investigator to set his levels precisely where he wishes them to be, the corre- 
lational study to some extent imposes values for different levels of a charac- 
teristic upon the investigator. Moreover, if the experimental investigator 
chooses to study the effects of two or more characteristics in the same study, 
the passages used can be made to reflect all combinations of characteristics 
and levels. It is just not possible to select passages from existing written 
material which guarantee that all combinations of characteristics and levels 
will be represented. Hence, it is possible to systematically evaluate the 
interaction between characteristics in experimental studies, and not possib-e to 
do so in correlational studies. 

There have been few experimental studies dealing with difficulty as 
a function of characteristics of written material. In addition to those described 
in this report, certain studies by Coleman (8, 9) serve as examples. 

Research on Language Development . The typical study of language devel- 
opment of school-age children employs subjects at different ages or school 
grades. Most studies of this kind have begun by obtaining writing samples of 
some length— for example, 1000 words— from the subjects. Typically, the sub- 
ject is required to write a theme on a topic of his choosing. The grammatical 
characteristics of the resulting material are then evaluated. The final step is 
to isolate those grammatical characteristics of writing— for example, clause 
length, proportion of all words that are adjectives— which follow from the pre- 
conception that physical (chronological) growth in the child should be accom- 
panied by linguistic growth. Such studies suggest variables worth considering 
in experimental research on the difficulty of written material. 

The language development literature of the sort described has been 
reviewed by McCarthy ( 10 ) and by Harrell ( 1 1 ). Many of the earlier studies- 
like those of readability research— have been concerned with such factors as 
sentence length, clause length, the ratio of subordinate clauses to all clauses, 
and ratios between the number of words in one grammatical class to those of 
another (as in the ratio of adjectives to verbs). 

Hunt (12) recently published a study of language development in school- 
age children which takes advantage of tools of analysis that the more modern 
grammars invite. Hunt employs the distinction made in modem grammars 
between surface components and deep components. In consequence, he was 
able to study the effects of age (school grade) on the number of kernel sentences 1 
produced or the number of sentence-combining transformations represented 
in a single-clause or multiple -clause construction. What is interesting about 
such factors is that, in the context of Hunt's study, these more sophisticated 
factors tended to covary with school grade more closely than did such surface 
factors as number of sentences in the 1000-word passage. 

‘As a convenience to the reader, a glossary of some technical, linguistic, and mathematical terms, defined 
as they are used in this report, is presented on p* iv* 
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If the maturation of school-age children is accompanied by increasing 
linguistic sophistication in written expression, perhaps it is also accompanied 
by an analogous shift in reading habits, and hence, in turn, to understanding 
written material. When reading, what are the units of grammar— the word, the 
phrase, the clause— to which one attends and do these units vary with age? An 
initial interest in such questions is reflected in articles by Baldwin and Baum (13) 
and Suci (14), who are members of the Reading Research Group at Cornell 
University. The preliminary work at Cornell suggests that the magnitude of the 
unit of grammar attended to when reading increases with age-grade. It seems 
worth keeping in mind that the grammatical units that prove to be basic to read- 
ing will also be basic to understanding. 



CHARACTERISTICS OF THE PRESENT STUDIES 

The present studies, which were conducted during the period from July 1964 
to October 1965, are concerned with discovering dimensions underlying the 
complexity of written material and determining functional relations between 
these dim ensions and measures of proficiency following instruction. While 
memorizing and understanding are not the same process, memorization 
measures— which are quicker, more reliable, and easier to use than measures 
of understanding— were used in these experiments on the assumption that such 
measures would be predictive of understanding. 

Six studies encompassing 14 experiments are described. Each experiment 
utilized one of four classes of units of content— paragraphs, sentence lists, 
sentences, or word lists. One of three primary measures of proficiency was 
used — cumulative presentation time to a rote recall criterion, cumulative 
presentation to a free recall criterion, or fixed-interval trials to a free recall 
criterion. The independent variables of an experiment fell into one or more of 
four general classes— those of load (information density), length, packaging, 
and word frequency. 

Units of Content 



Pertinent characteristics of the four classes of units of content employed 
in the various experiments were as follows: 

(1) Paragraphs . Each unit of content was a paragraph consisting of 
two sentences written in a relaxed expository style. The second sentence of 
a paragraph was formally related to the first, usually on the basis of a rule of 
the form "A noun of Sentence 1 equals a pronoun of Sentence 2." Paragraphs 
were used as experimental content in one study. 

(2) Sentence lists . Each unit of content (that is, each list of sentences) 
consisted of a set of simple sentences — one- clause constructions containing 

a single kernel sentence. For present purposes, a "kernel sentence" will be 
taken as an irreducible sentence. Representative types of such sentences are 
"N is A" (e.g., "The shovel is dirty") and "N V NP N" (e.g., "The baker gave 
the cookie to the shortstop"). 1 The sentences of a set were arranged in a list, 
one below the other. 

Sentence lists were used in two studies, containing four experi- 
ments. All sentences of all lists used in an experiment were of the same kernel 

‘In all content formulas used in this report, N=noun, A=adjective, V=verb, and P = preposition. 



